A Framework for Generating Distributed-Memory Parallel Programs for Block Recursive Algorithms

and implement high-performance algorithms to compute the discrete Fourier transform (DFT) [17, 24] and matrix multiplication [14, 15] on shared-memory vector multiprocessors. The significance of the tensor product lies in its ability to model both the computational structures occurring in block-recursive algorithms and the underlying hardware structures, such as the interconnection networks [19, 20]. The goal of this paper is to develop a framework for synthesizing efficient distributed-memory programs for block recursive algorithms. In this framework we start with a mathematical specification of the computation using tensor products. We present techniques for developing efficient message passing codes by analyzing the structure of the input mathematical formulas. Designing and implementing an algorithm using the tensor product involves viewing the computation as a linear transformation, rewriting the linear transformation as a matrix product of tensor products of smaller computation matrices, recursively applying the rewriting rule to smaller computation matrices, and translating the components of the resulting tensor product formula into vector/parallel operations, iterative loops, and data movement operations. Once expressed using the tensor product notation, several forms of the algorithm with different performance characteristics can be derived by exploiting the algebraic properties of its matrix representation. In this paper, we provide a framework for designing and implementing programs for distributed-memory MIMD machines. We first present an algebraic representation based on the tensor product for describing the semantics of regular data distributions. Using this algebraic representation, we develop criteria for identifying data distributions which will permit a communication-free implementation of the computation represented by a given tensor product formula. Using these criteria we develop techniques for synthesizing programs for distributed-memory machines with the goal of minimizing the communication overhead. This leads to generation of programs under two different programming models. These two models are based on difJOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING 34, 137–153 (1996) ARTICLE NO. 0051